System and method for human computer interaction

ABSTRACT

The present invention relates to the technical field of the human computer interaction, more particularly to, a system and method for human computer interaction. The system for human computer interaction comprises a projection unit, a first image sensing unit, a second image sensing unit, an interface unit, an image processing unit, a projected interface processing unit, and a controlling unit. The system for human computer interaction provided by the present invention may easily project the human computer interaction interface on all kinds of planes encountered in people&#39;s daily life, to realize display of the human computer interaction interface everywhere, and improve users&#39; experience.

TECHNICAL FIELD

The present invention relates to the technical field of human computerinteraction, particularly to a system and method for human computerinteraction.

BACKGROUND

One important aim for people is to work, have fun and amusement by usingelectrical devices such as computers, mobile phones etc. convenientlyand comfortably in anywhere, that is omnipresent human computerinteraction. However, the existing solution does not achieve goodresults, on the one hand, though the prior mobile phone has strongcalculating and processing ability, the size of the phone is small,resulting in the display screen is too small to be suitable forwatching, and the keyboard is too small to be operated conveniently byusers, on the other hand, though the prior notebook may be portablycarried, by comparing with the mobile phone, it still seems cumbersomeand can't be carried conveniently, resulting from the size is much largeand the weight is much heavy. If the notebook is designed smaller, thesame situation as the phone that the notebook is not used convenientlywill happen.

In addition, most of the prior electrical devices have an externaloperating and controlling panel, a display panel, and a keyboard or amouse etc., which will actually have big influence on the reliabilityand stability of the devices, such as electrical devices on the vessels(which need strong resistance on salt, fog and humidity etc.,).Meanwhile, the size of the devices will become big because of theexternal operating and controlling panel, and the display panel.

For the prior portable or wearable human computer interaction systemusing projection, one typical manner is to constitute a human computerinteraction system with a depth sensor, which is formed by using aprojection-invisible optical projection unit with infrared structuredlight and an infrared image sensing unit, and a visible light projectoretc. (some systems even include a visible light image sensing unit),however, the depth sensing unit (such as kinect of Microsoft) utilizedby such system normally has big power consumption and volume, especiallyfor an application of the smart phone, its size gets bigger because itneeds adding infrared projection and infrared image sensing unitadditionally in the phone. The other manner is to constitute a portablehuman computer interaction system which is formed by one visible lightprojecting unit and one visible light image sensing unit, however, suchsystem needs producing invisible structured light for obtaining theoperation of touching the projecting plane by fingers accurately, andthe principle of method for producing the invisible structured light isgenerally to require projecting frequency of the projecting unit biggerthan 75 Hz, normally 120 Hz, as well as frame rate for image collectingof the image sensing unit is same as frame rate for projecting,therefore the power consumption of such system would be excessive toobecause the frame rate for projecting and image collecting is very high.In addition, the computation complexity of such system will be enhancedresulting from that the invisible structured light still has coding anddecoding.

In summary, the prior electrical devices have the following defects, (1)size of the operating and controlling panel or the display panel is toosmall to be suitable for watching and operating, (2) the volume andweight of the electrical devices with the operating and controllingpanel or the display panel is too big to be portably carried, (3)reliability and stability of the electrical devices with the operatingand controlling panel or the display panel is too weak to work undersome special severe environments, and (4) both power consumption andsize of the prior portable or wearable human computer interaction systemwith projection is too high and big.

SUMMARY OF THE INVENTION Technical Problem to be Solved

The technical problem to be solved by the present invention is how toprovide a system for human computer interaction which is miniaturizedand constituted based on double image sensing units and a projectingunit, and its method, to achieve that a human computer interactionelectrical device is used conveniently and comfortably in anywhere, sothat such electrical device can work in some special severeenvironments.

Technical Solution

In order to solve the above mentioned problem, a system for humancomputer interaction is provided, which comprises a projecting unit, afirst image sensing unit, a second image sensing unit, an interfaceunit, an image processing unit, a projected interface processing unit,and a controlling unit, in which,

the projecting unit is for receiving human computer interactioninterface information data output by the interface unit, and projectingand outputting the data to a projecting plane;

the first image sensing unit and the second image sensing unit is forcollecting projected image information on an area of the projectingplane, and outputting to the image processing unit through the interfaceunit;

the interface unit is coupled to the projecting unit, the first imagesensing unit, the second image sensing unit, the image processing unit,the projected interface processing unit and the controlling unit is forsynchronous controlling the first image sensing unit, the second imagesensing unit and the projecting unit;

the image processing unit is for receiving image data collected by thefirst image sensing unit and the second image sensing unit; receiving ahuman computer interaction interface image output by the projectedinterface processing unit, and extracting feature information in thehuman computer interaction interface image; calculating and obtainingsystem inherent internal parameters which comprise aberration distortionparameters, focus, pixel size and center point position on optical lensof the first image sensing unit, the second image sensing unit and theprojecting unit, storing the system inherent internal parameters on anon-volatile memory, and outputting the system inherent internalparameters to the interface unit; extracting corresponding featureinformation for the image data, and matching it with the featureinformation output by the projected interface processing unit;calculating and obtaining system inherent external parameters whichcomprise position and gesture relationship among the first image sensingunit, the second image sensing unit and the projecting unit; andcalculating and obtaining system variable external parameters whichcomprise position and gesture relationship between the human computerinteraction system and the projecting plane, and outputting position andgesture relationship parameters between the projecting plane and theprojecting unit in the obtained system variable external parameters tothe projected interface processing unit;

the image processing unit is further for processing images of aprojection area, which is collected by the first image sensing unit andthe second image sensing unit, obtained through the interface unit,searching a fingertip or other touch element according to apredetermined fingertip feature, and obtaining a distance between thefingertip or other touch element and the projecting plane based on aprinciple of binocular disparity, if the distance is in the range of apredetermined first threshold value, determining a touch operation isoccurred and outputting touch information;

the projected interface processing unit is for receiving human computerinteraction information intended to be projected, extracting featureinformation from the human computer interface interaction information,and outputting the feature information to the image processing unit; theprojected interface processing unit is further for receiving theposition and gesture parameters between the projecting plane and theprojecting unit in the system variable external parameters output by theimage processing unit, performing shape predistortion correctionprocessing to interface images in the input human computer interactioninterface information based on the parameter and then outputting to theinterface unit;

the controlling unit is for controlling all units of the system, andcoordinately controlling work of all units of the system; thecontrolling unit is further for controlling the human computerinteraction system in the statuses of obtaining the system inherentinternal or external parameters, obtaining the system variable externalparameters, and obtaining a fingertip and touch operation.

Preferred, the interface unit includes an image sensing interface moduleand a projecting interface module, wherein, the image sensing interfacemodule is for receiving the system inherent internal parameters outputby the image processing unit; receiving image data from the first imagesensing unit and the second image sensing unit, and outputting to theimage processing unit after performing optical distortion correction ofthe image data; the projecting interface module is for receiving theprojected image data output by the projected interface processing unit,performing optical aberration predistortion correction processing of theprojected interface images output by the projected interface processingunit based on the system inherent internal parameters, and outputtingthe corrected projected interface images to the projecting unit.

Preferred, the image processing unit includes a system inherentinternal-external parameters obtaining module, a feature extractingmodule, a fingertip obtaining module, and a fingertip touch determiningmodule, wherein,

the system inherent internal-external parameters obtaining module is forreceiving the system inherent internal or external parameters output bythe feature extracting module, and storing the parameters in anon-volatile memory; outputting the system inherent internal parametersto the projecting interface module and the image sensing interfacemodule; and outputting the system inherent internal or externalparameters stored in the non-volatile memory to the feature extractingmodule;

the feature extracting module is for receiving feature information ofhuman computer interaction interface output by the projected interfaceprocessing unit, under the state where the human computer interactionsystem is obtaining the system inherent internal-external parameters,and receiving images output by the image sensing interface module inaccordance with the feature information, extracting the correspondingfeature information of three from images of the projection areacollected by the first image sensing unit and the second image sensingunit, performing matching of the feature information, calculating andobtaining the system inherent internal-external parameters, andoutputting the system inherent internal-external parameters to thesystem inherent internal-external parameters obtaining module;

the feature extracting module is for receiving feature information ofhuman computer interaction interface output by the projected interfaceprocessing unit, under the state where the human computer interactionsystem is obtaining the system variable external parameters, receivingimages output by the image sensing interface module in accordance withthe feature information, extracting the corresponding featureinformation of three from images of the projection area collected by thefirst image sensing unit and the second image sensing unit, performingmatching of the feature information, calculating and obtaining thesystem variable external parameters, receiving the system inherentinternal-external parameters output by the system inherentinternal-external parameters module, respectively calculating andobtaining homography matrixes between the first image sensing unit andthe projecting unit and between the second image sensing unit and theprojecting unit, and outputting the matrixes to the fingertip touchdetermining module;

the fingertip obtaining module is for receiving image data output by theimage sensing interface module, under the state where the human computerinteraction system is obtaining a fingertip and touch operation, andprocessing the image data to obtain position coordinates information forthe fingertip of the finger or front end of touch elements in imageplanes of the two image sensing units respectively, and outputting theposition coordinates information to the fingertip touch determiningmodule;

the fingertip touch determining module is for receiving the positioncoordinates information and the homography matrixes, under the state ofobtaining a fingertip and touch operation; calculating a distancebetween the fingertip and the first image sensing unit and a distancebetween the fingertip and the second image sensing unit with theprinciple of binocular disparity, and, calculating and obtaining adistance between the projecting plane and the fingertip, or the frontend of the other touch element, by means of position and gesturerelationship between the first image sending unit and the projectingplane and between the second image sensing unit and the projectingplane;

preferred, if the distance between the projecting plane and thefingertip, or between the projecting plane and the front end of theother touch element is smaller than the predetermined first thresholdvalue, then touch operation is determined; if the touch operation isoccurred, the fingertip touch determining module is further forcalculating and obtaining coordinates position of the touch points inthe human computer interaction interface originally projected inaccordance with the homography matrixes, and outputting the fingertiptouch information.

Preferred, the projected interface processing unit includes an interfaceprocessing module and an interface inputting module;

wherein, the interface inputting module is for receiving human computerinteraction interface information data input by the outside, andoutputting to the interface processing module;

the interface processing module is for receiving the human computerinteraction interface information data output by the interface inputtingmodule, receiving the relative position and gesture parameters betweenthe projecting plane and the projecting unit from the system variableexternal parameters, and performing image geometric predistortionprocessing on the human computer interaction interface images, andoutputting it to the interface unit;

the interface processing module is further for extracting featureinformation of the human computer interaction interface from the inputhuman computer interaction interface information, and outputting thefeature information of the human computer interaction interface to theimage processing unit.

Preferred, the system further comprises a feature deformation measuringmodule, wherein

the controlling unit is for coordinating the human computer interactionsystem into the state of fingertip coarse positioning or fingertip finepositioning,

if the human computer interaction system is in the state of fingercoarse positioning, the controlling unit controls and selects anyone ofthe first image sensing unit and the second image sensing unit into aresting state, or stops all modules in the image processing unitprocessing images collected by the selected image sensing unit, or,selects the nearest one between the first image sensing unit and thesecond image sensing unit from the projecting unit into a resting stateor stops all modules in the image processing unit processing imagescollected by the selected image sensing unit;

if the human computer interaction system is in the state of fingertipfine positioning, the controlling unit controls both the first imagesensing unit and the second image sensing unit into a normal workingstate, or controls to start all modules in the image processing unitprocessing of images collected by the first image sensing unit and thesecond image sensing unit;

if the human computer interaction system is in the state of fingertipcoarse positioning, the feature extracting module calculates and obtainsthe homography matrix between the projecting unit and the image sensingunit in the working state based on the system variable externalparameters, receives the feature information of the human computerinteraction interface output by the interface processing module, orreceives human computer interaction interface image intended to beprojected output by the projected interface processing unit, andextracts the feature information of the human computer interactioninterface images through the feature extracting module; performscoordinate conversion to the feature information with the homographymatrix, into a coordinate system of an image plane of the image sensingunit in the working state, and obtains corresponding feature informationcombined by a plurality of first basis feature units; obtains featureinformation combined by a plurality of second basis feature units in thehuman computer interaction interface on the projecting plane collectedby the image sensing unit in the working state, the feature extractingmodule performs matching to the feature information combined by theplurality of first basis feature units and that combined by theplurality of second basis feature units in a coordinate system of animage plane of the image sensing unit or in a coordinate system of animage plane of the projecting unit, to obtain matchable basis featureunit pairs, and outputs the matchable result to the feature deformationmeasuring module;

the feature deformation measuring module receives matchable result ofthe feature information output by the feature extracting module,performs comparing to all matchable basis feature units, respectivelycalculates variation between the basis feature unit pairs, which atleast includes shape variation or area variation between the basisfeature unit pairs, or offset of the position coordinates of featurepoints; calculates a distance from the finger to the projecting plane atthe point of deformation of the second basis feature unit based on thevariation between the basis feature unit pairs, and evaluates the areafor the finger present in the image plane of the first image sensingunit and the second image sensing unit, if the distance from the fingerto the projecting plane at the point of deformation of the second basisfeature unit is smaller than the predetermined first threshold value,touch operation is directly occurred; if the area of the touching areais too large, outputs coordinates of the possible touching area to thefingertip obtaining module to obtain accurate touching coordinates,otherwise, directly outputs coordinates of the corresponding touchingarea to the fingertip touch determining module, if the distance from thefinger to the projecting plane at the point of deformation of the secondbasis feature unit is exceed the predetermined second threshold value,no processing is performed by following, if the calculated distance fromthe finger to the projecting plane at the point of deformation isbetween the first and the second threshold value, outputs thecoordinates of the finger evaluated in the area in the image plane ofthe first image sensing unit and the second image sensing unit; whereinthe predetermined first threshold value is smaller than thepredetermined second threshold value;

the fingertip obtaining module is for receiving possible presenting areaof the fingertip in the human computer interaction interface images,which are collected by the first image sensing unit and the second imagesensing unit, output by the feature deformation measuring module, andrespectively searching the fingertip only in the two corresponding area,then outputting two-dimensional coordinates of the fingertiprespectively on the image planes collected by the first image sensingunit and the second image sensing unit to the fingertip touchdetermining module;

the fingertip touch determining module is further for receivingfingertip touch information, which includes accurate coordinatesinformation of fingertip touch position and position information offingertip touching area, output by the feature deformation measuringmodule;

when receiving touch operation output by the feature deformationmeasuring module and the accurate fingertip position coordinates of theimage planes of the image sensing units output by the fingertipobtaining module, if the distance is smaller than the predeterminedfirst threshold value, calculating and obtaining accurate position whichthe finger touched the human computer interaction interface originallyprojected in accordance with the homography matrixes between theprojecting unit and the first image sensing unit and/or the second imagesensing unit, and outputting the fingertip touch information.

Preferred, the projected interface processing unit receives graphics orimage information of the human computer interaction interface input bythe outside, or further receives feature information extracted from thehuman computer interaction interface, which includes types of the basisfeature unit having points, lines, planes, letters or regular graphics,or images, and position information of the basis feature unit in thehuman computer interface images;

the interface processing module further directly extracts featureinformation data in the interface from the human computer interactioninformation;

the human computer interaction interface information of the interfaceinputting module further includes ranges of every clickable area in thehuman computer interaction interface;

the interface processing module further extracts the ranges of everyclickable area in the human computer interaction interface and outputsit to the feature deformation measuring module;

the feature deformation measuring module further receives the ranges ofevery clickable area in the human computer interaction interface fromthe interface processing module, when the system is in the state offingertip coarse positioning, determines whether evaluated positionrange of the finger of the coarse positioning is in the range inaccordance with the ranges of every clickable area, if it is, thendirectly outputs touch information to the fingertip touch determiningmodule; if not, then outputs evaluated position ranges of the finger ofthe coarse positioning to the fingertip obtaining module, and sendsfingertip touch signals to the fingertip touch determining module.

Preferred, the interface processing module is further for producingimages with different intensity or colors, to output them to theprojecting interface module and the feature extracting module is forobtaining projecting color predistortion processing function; theinterface processing module is further for receiving images withdifferent intensity or colors from the outside of the system, to outputthem to the projecting interface module and the feature extractingmodule is for obtaining projecting color predistortion processingfunction; the interface processing module is for receiving projectingcolor predistortion processing function f⁻¹ or a lookup table output bythe feature extracting module, the interface processing module isfurther for performing color aberration pre-correction processing to theinput human computer interaction interface, to realize colors of thehuman computer interface projected on the projecting planes do notdistort;

the feature extracting module is further for receiving images withdifferent intensity or colors output from the interface processingmodule, further for receiving corresponding images with differentintensity or colors collected by the first image sensing unit and thesecond image sensing unit, and performing the images comparingprocessing, to obtain difference analysis between images intended to beprojected by the system and projected images of the projecting planes,to obtain optical modulation property of the projecting planes, andobtain projecting color predistortion processing function f⁻¹ or alookup table in accordance with the optical modulation property, thefeature extracting module outputs the obtained projecting colorpredistortion processing function f⁻¹ or the lookup table to theprojecting interface processing module.

Preferred, the controlling unit is further for starting the featureextracting module to obtain the system variable external parameters atregular time, that is obtain the relative position and gestureinformation of the projecting plane relative to the system;

the feature extracting module is further for comparing the currentobtained system variable external parameters to the earlier obtainedsystem variable external parameters, if the current obtained systemvariable external parameters is different from the earlier obtainedsystem variable external parameters, outputting the varied systemvariable external parameters to the interface processing module again,and further for calculating the homography matrix between the projectingunit, and the first and second image sensing unit, and outputting to thefingertip touch determining module.

A method for human computer interaction performed by the human computerinteraction system is also provided by the present invention, itcomprises:

S1, initially calibrating the human computer interaction system, toobtain system inherent internal and external parameters;

S2, obtaining system variable external parameters, and calculatinghomography matrixes respectively between the projecting unit, and thefirst and second image sensing unit based on the system variableexternal parameters and the system inherent external parameters;

S3, finely obtaining, by the fingertip obtaining module, positioncoordinates of a fingertip on planes of images collected by the firstimage sensing unit and the second image sensing unit and calculating adistance from the fingertip relative to a projecting plane, andsearching the fingertip, by the feature extracting unit, from imagesreceived from and collected by the first image sensing unit and thesecond image sensing unit, and outputting the obtained positioncoordinates of the fingertip in the image planes of the first imagesensing unit and the second image sensing unit to the fingertip touchdetermining module;

S4, calculating, by the fingertip touch determining module, a positionrelationship of the fingertip relative to the image sensing units,utilizing the principle of binocular disparity, in accordance with thereceived position coordinates of the fingertip respective in the planesof the first image sensing unit and the second image sensing unit, andcalculating the relative position and gesture between the image sensingunits and the projecting plane based on the system variable externalparameters, and calculating and obtaining the distance of the fingertipfrom the projecting plane,

if a distance between the fingertip and the projecting plane is smallerthan a predetermined first threshold value, determining touch operationis occurred, and then turning to step S5; otherwise, returning to stepS3;

S5, outputting touch information, and calculating and obtaining accurateposition coordinates of the touching points in the coordinates system ofthe human computer interaction interface, in accordance with thecoordinates of the touch points in the image planes of the first imagesensing unit and/or the second image sensing unit combined withhomography matrixes respectively between the projecting unit, and thefirst image sensing unit and/or the second image sensing unit, and thenoutputting them.

Preferred, the step S3 further comprises: S31, controlling, by thecontrolling unit, the human computer interaction system into the stateof fingertip coarse positioning, selecting anyone image sensing unitinto the state of resting from the first image sensing unit and thesecond image sensing unit, or stopping all modules of the imageprocessing unit processing to the images collected by the selected imagesensing unit; or selecting the nearest one distant from the first imagesensing unit and the second image sensing unit to the projecting unitinto the resting state, or stopping all modules of the image processingunit processing to images collected by the selected image sensing unit,to further obtain, by the feature extracting module, some basis featureunit pairs matched by a first basis feature unit and a second basisfeature unit, to further calculate and obtain variation between thebasis feature unit pairs utilizing the feature deformation measuringmodule, for performing rough determination of the possible area positionthat the fingertip presents in the collected images of the image sensingunit selected to work, which comprises:

If a distance from the finger at the points of the deformation of thesecond basis feature unit to a projecting plane is smaller than apredetermined first threshold value, determining a touch operation isoccurred; if touching area is too large, then outputting coordinates ofthe touching area to the fingertip obtaining module, to obtain accuratetouching coordinates; otherwise, directly outputting positioncoordinates of the corresponding touch area to the fingertip touchdetermining module, then turning to step S5;

If a distance from the finger at the points of the deformation of thesecond basis feature unit to a projecting plane is bigger than apredetermined second threshold value, determining a touch operation isnot occurred, no process is performed by following, and returning tostep S31;

If a distance from the finger at the points of the deformation of thesecond basis feature unit to a projecting plane is between apredetermined first and a second threshold value, outputting areacoordinates of the fingertip evaluated in the image planes of the firstimage sensing unit and the second sensing unit to a fingertip obtainingmodule, then turning to step S32;

S32, finely searching a fingertip, in accordance with the coarsepositioning area of the finger obtained by step S31, this step includessearching, by the fingertip obtaining module, the fingertip in the areaof the finger coarse positioning in the images received from andcollected by the first image sensing unit and the second image sensingunit, and outputting the position coordinates of the fingertip obtainedin the image planes of the first image sensing unit and the second imagesensing unit to the fingertip touch determining module, then turning tostep S4.

Preferred, the step S2 further comprises scanning and obtaining aprojecting color predistortion parameter of the projecting plane andperforming color predistortion correction. Preferred, the step S2further includes starting the feature extracting module to obtain systemvariable external parameters at regular time;

that is, obtaining relative position and gesture information of theprojecting plane relative to the system, if the relative position andgesture is changed, re-calculating homography matrixes between the firstimage sensing unit and the second image sensing unit, and the projectingunit, and outputting again the changed system variable parameters to theinterface processing module, which updates the system variableparameters, and performs the processing of projecting shapepredistortion based on the changed system variable parameters, torealize tracking the projecting plane in real time.

Effect

A new system and method for human computer interaction, based on twoimage sensing units and one projecting unit, is provided by the presentinvention, it achieves the following beneficial effect:

1, accurately projecting the human computer interaction interface on allkinds of planes in people's daily life through the projecting unit, toachieve display of the human computer interaction interface in anywhere;

2, accurately obtaining the distance between the fingertip of the userand the projecting plane with the principle of binocular disparity, whenthe user is performing human computer interaction operation in the areawithout obvious feature in the human computer interaction interface;

3, further determining the possible area which the fingertip may presentin the projected interface area, by utilizing that the basis featureunit in the interface will have deformation or the basis feature unitwill have offset of the position coordinates in the images of the humancomputer interface when the users' hand is present in the area of theprojected interface, if there is obvious feature information availablein the projected human computer interaction interface; greatly narrowingthe searching scope of the fingertip, lowering the computationcomplexity of the system, and improving real-time performance of thesystem, meanwhile, this procedure merely needs one image sensing unitand one projecting unit to be operated, to further lower the powerconsumption of the system;

4, the requirement for frame rate for collecting of two image sensingunits is not high because projecting and collecting of invisiblestructured light is not needed, the frame rate for image collecting isto be guaranteed that an image is collected when the finger clicks onthe interface every time, while the speed of clicking the interface byhands is very slow, for example, the rate that one single finger clickson the interface over three times per second is very high;

5, the present invention can support the human computer interactioninterface information input to the interface inputting module alsoincludes feature information of the human computer interaction interfaceintended to be projected except the image of the human computerinteraction interface, and firstly calculate the feature informationutilizing the known interface to be projected, and the interfaceprocessing module do not need extracting feature information from theinput human computer interface images, so that the computationcomplexity and power consumption of the interface processing module canbe greatly lowered; additionally, the human computer interactioninterface information input to the interface processing module furtherinclude a range of every possible clickable area, such as the range ofarea of a button, an icon, a menu etc., in the human computerinteraction interface, therefore, when calculating these area where thefingertip touched, the position of the touch points in the humancomputer interaction interface is not needed to be accuratelycalculated, so that in practical applications, the coarse positioning ofthe fingertip can realize calculating of the position area of the touchpoints of the fingertip in many cases, thereby the computationcomplexity and power consumption can be saved;

6, sine the disclosed human computer interaction system can work withoutan operating and controlling panel and a display panel, its reliabilityand stability is greatly improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a structure schematic drawing illustrating a human computerinteraction system according to an embodiment 1 of the presentinvention;

FIG. 2 is a structure schematic drawing illustrating a human computerinteraction system according to an embodiment 2 of the presentinvention;

FIG. 3 is a schematic drawing illustrating a single window humancomputer interaction interface according to an embodiment of the presentinvention;

FIG. 4 is a schematic drawing illustrating a multiwindow human computerinteraction interface according to an embodiment of the presentinvention;

FIG. 5 is a schematic drawing illustrating a virtual keyboard accordingto an embodiment of the invention;

FIG. 6 is a flowchart illustrating a method for human computerinteraction according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments of the present invention will be detailed describedaccompanying with the embodiments and drawings. The exemplaryembodiments are for illustrating the present invention, but not forlimiting the scope of the present invention.

Embodiment 1

FIG. 1 is a structure schematic drawing illustrating a human computerinteraction system according to an embodiment 1 of the invention; asshown in FIG. 1, the system for human computer interaction disclosed bythe present invention comprises two image sensing units (the imagesensing unit 1 (or the first image sensing unit) and the image sensingunit 2 (or the second image sensing unit)), one projecting unit, aninterface unit, an image processing unit, a projected interfaceprocessing unit, and a controlling unit etc. The system receives humancomputer interface information through the projected interfaceprocessing unit, and performs processing to the interface informationand after that, sends to the projecting unit for projecting on somenormal planes such as desks, walls, paper, hand palms, arms etc., toaccurately project and display the human computer interface, and thenthe user may perform bare hand operation similar to touching the screenon the projected human computer interface, while the two image sensingunits collect the images on the projection area, identify the operationfor human computer interaction, and output the corresponding informationof the interaction operation, thereinto, the key point is to identifywhether the hand touches the projected interface, and the position ofthe touch point on the projected human computer interface, here, acontrolling unit coordinately controls work of all other units of thesystem.

The projecting unit is for projecting human computer interactioninterface information data received from and output by the projectinginterface module, and projecting and outputting to a projecting plane;

The image sensing unit 1 and the image sensing unit 2 are for collectingprojecting image information on an area of the projecting plane, andoutputting to the image processing unit through the interface unit;

The interface unit is coupled to the projecting unit, the image sensingunits 1 and 2, the controlling unit, the projected interface processingunit and the image processing unit, for receiving system inherentinternal parameters, (including optical distortion parameters of theimage sensing unit 1, the image sensing unit 2 and the projecting unit),output by the image processing unit; the interface unit receives theimage data from the image sensing units 1 and 2, and performs opticaldistortion correction, and after that, outputs to the image processingunit, as well as receives the projected image data output by theprojected interface processing unit and performs optical distortioncorrection, and then outputs to the projecting unit; the interface unitmainly includes an image sensing interface module and a projectinginterface module; the image sensing interface module is for receivingthe optical distortion parameters of the image sensing units 1 and 2from the system inherent internal parameters output by the imageprocessing unit; the image sensing interface module is further forreceiving image data from the image sensing unit 1 and the image sensingunit 2, and respectively performing optical distortion correction toimages collected by the image sensing units 1 and 2, based on theoptical distortion parameters of the image sensing units 1 and 2 in thesystem inherent internal parameters, to eliminate the optical aberrationdistortion brought by properties of the optical components of the imagesensing units, and then outputting the corrected images to the imageprocessing unit; the projecting interface module is for receivingprojecting data from the projected interface processing unit, performingoptical aberration predistortion correction processing of the images ofthe interface to be projected output by the projected interfaceprocessing unit based on the optical aberration parameters of theprojecting unit in the system inherent internal parameters, to eliminatethe optical aberration distortion brought by properties of the opticalcomponents of the projecting unit, and outputting the corrected image ofthe interface to be projected to the projecting unit. The interface unitis further for synchronous controlling the image sensing unit 1, theimage sensing unit 2, and the projecting unit.

The image processing unit is for receiving image data of the imagesensing unit 1 and the image sensing unit 2, output by the image sensinginterface module, receiving and extracting the human computerinteraction interface image output by the projected interface processingunit and feature information of the image (which at least includes typesof a basis feature unit such as points, lines or other regular graphicsor images etc., and further includes the position coordinates of thebasis feature unit in the human computer interaction interface images);The image processing unit is for calculating and obtaining systeminherent internal parameters (which comprise aberration distortionparameters, focus, pixel sizes and center point's position etc., onoptical lens of the two image sensing units and the projecting unit),storing the system inherent internal parameters on a non-volatilememory, and outputting the system inherent internal parameters to theinterface unit; The image processing unit is extracting correspondingfeature information of the received image data, and matching it with thefeature information output by the projected interface processing unit;calculating and obtaining position and gesture relationship among theimage sensing unit 1, the image sensing unit 2 and the projecting unit(which is called system inherent external parameters) and position andgesture relationship between the human computer interaction system andthe projecting plane (which is called system variable externalparameter); The image processing unit is further for outputting theposition and gesture relationship parameters between the projectingplane and the projecting unit in the obtained system variable externalparameters to the projected interface processing unit; The imageprocessing unit is further for processing the images, which arecollected by the image sensing units 1 and 2, of the projection area,obtained through the interface unit, searching the fingertip (or othertouch element) operated by the user based on fingertip feature, andobtaining the distance from the fingertip (or other touch element) tothe projecting plane based on the principle of binocular disparity, ifthe distance is in the range of some one threshold value, determiningthe touch operation is occurred, and outputting the touch information,which at least includes the position coordinates of the touch points inthe projected human computer interaction interface;

The projected interface processing unit is for receiving human computerinteraction interface information intended to be projected, which atleast includes human computer interaction Interface graphics or imagesintended to be projected, input from the outside; The projectedinterface processing unit is further for extracting obvious featureinformation from the input human computer interaction interfaceinformation, and outputting the feature information to the imageprocessing unit; The projected interface processing unit is further forreceiving relative position and gesture parameters between theprojecting plane and the projecting unit in the system variable externalparameters output by the image processing unit, and performing shapepredistortion correction processing to interface images of the inputhuman computer interaction interface information based on such parameterand then outputting to the interface unit;

The controlling unit is for controlling all units of the system, andcoordinately controlling work of all units of the system; thecontrolling unit is further for controlling the human computerinteraction system in the statuses of obtaining the system inherentinternal or external parameters, obtaining the system variable externalparameters, obtaining a fingertip and touch operation.

The projected interface processing unit further includes an interfaceprocessing module and an interface inputting module;

The interface inputting module is for receiving human computerinteraction interface information data, (which may be compressed oruncompressed), input by the outside, decompressing the compressed humancomputer interaction interface information and outputting to theinterface processing module; the interface inputting module may furtherinclude wireless interfaces, or wired interfaces.

The interface processing module is for receiving the human computerinteraction interface information data output by the interface inputtingmodule, and receiving the relative position and gesture parametersbetween the projecting plane and the projecting unit from the systemvariable external parameters output by the feature extracting module ofthe image processing unit; The interface processing module is furtherfor performing geometric predistortion processing to the received humancomputer interaction interface, and outputting it to the interface unit;The interface processing module is further for extracting featureinformation of the human computer interaction interface from the inputhuman computer interaction interface information, and outputting thefeature information to the image processing unit;

The image processing unit includes a system inherent internal-externalparameters obtaining module, a feature extracting module, a fingertipobtaining module, and a fingertip touch determining module, wherein,

The system inherent internal-external parameters obtaining module is forreceiving the system inherent internal or external parameters output bythe feature extracting module, and storing the parameters in anon-volatile memory; The system inherent internal-external parametersobtaining module is for outputting the inherent internal parameters tothe projecting interface module and the image sensing interface moduleof the interface unit; and outputting the inherent internal or externalparameters stored in the non-volatile memory to the feature extractingmodule;

The feature extracting module is for receiving the human computerinteraction interface feature information output by the projectedinterface processing unit, under the state of obtaining the systeminherent internal-external parameters controlled by the controllingunit, in accordance with characteristic of the feature information,extracting the corresponding feature information of three from images ofthe projection area, which is collected by the image sensing unit 1 andthe image sensing unit 2, output by the image sensing interface module,and performing matching of the feature information, calculating andobtaining the system inherent internal-external parameters, andoutputting to the system inherent internal-external parameters obtainingmodule; The feature extracting module is for receiving the humancomputer interaction interface feature information output by theprojected interface processing unit, under the state of obtaining thesystem variable external parameters controlled by the controlling unit,in accordance with characteristic of the feature information, extractingthe corresponding feature information of three from images of theprojection area, which is collected by the image sensing unit 1 and theimage sensing unit 2, output by the image sensing interface module, andperforming matching of the feature information, calculating andobtaining the system variable external parameters, that is the relativeposition and gesture between the projecting plane and the image sensingunits 1 and 2, and between the projecting plane and the projecting unit;the feature extracting module is further for receiving the systeminherent internal-external parameters output by the system inherentinternal-external parameters obtaining module, calculating and obtaininghomography matrixes (that is, conversion relationship of the coordinatesbetween any points among two planes of the image sensing units and theplane of the projecting unit) between planes of the two image sensingunits, (that is two imaging planes of the image sensor chipsrespectively,) and the plane of the projecting unit, (that is the planeof the projecting chip), combing with the obtained system variableexternal parameter and outputting the matrixes to the fingertip touchdetermining module;

The fingertip obtaining module is for receiving image data of the imagesensing unit 1 or the image sensing unit 2 output by the image sensinginterface module, and through processing the image data collected by theimage sensing unit 1 and the image sensing unit 2, obtaining positioncoordinates information of the fingertip or front end of touch elementspositioned in image planes of the two image sensing units respectively,and outputting the position coordinates information to the fingertiptouch determining module;

The fingertip touch determining module is for receiving the positioncoordinates information of the fingertip or front end of touch elementspositioned in image planes of the two image sensing units respectivelyoutput by the fingertip obtaining module; further for receiving thesystem variable parameters output by the feature extracting module;further for receiving the homography matrixes between two planes of theimage sensing units and the plane of the projecting unit output by thefeature extracting module, The fingertip touch determining module iscalculating the distance between the finger and, the image sensing units1 and 2 with the principle of binocular disparity, and calculating andobtaining the distance between the projecting plane and the fingertip,(or the front end of the other touch element), by using position andgesture relationship between the image sending units 1 and 2, and theprojecting plane in the system variable external parameters; if thedistance is smaller than one predetermined threshold value, then touchoperation is determined as occurred; if the touch operation is occurred,the fingertip touch determining module is further for calculating andobtaining coordinates position of the touch points in the human computerinteraction interface originally projected in accordance with thehomography matrixes between the image sensing units 1 or 2 and theprojecting unit received from the feature extracting module, andoutputting the fingertip touch information, which at least includes theposition coordinates information which the fingertip touched the humancomputer interaction interface originally projected; The manner that thefingertip touch determining module outputs the touch information may bewireless method or wired method.

Several key points of the system will be described by the following.

1, Obtaining System Inherent Internal-External Parameters

The system needs obtaining the inherent internal parameters and externalparameters of the human computer interaction system constituted by twoimage sensing units and the projecting unit firstly before it leave thefactory or is initially used. The system inherent internal parametersinclude internal parameters of the image sensing units and theprojecting unit, which are mainly optical aberration parameters broughtby properties of their respective optical components. The systeminherent external parameters include the relative position and gesturerelationship among three of the image sensing units 1 and 2, and theprojecting unit. During the procedure of obtaining the system inherentinternal-external parameters, the controlling unit controls the wholesystem in the state of obtaining inherent internal-external parameters,firstly controls two image sensing units to respectively obtain theinternal parameters, in this embodiment, this procedure is performed bythe feature extracting module and the system inherent internal externalparameters obtaining module together, here the feature extracting modulereceives characteristic images intended to be calibrated (such as blackand white or colorful chessboard images printed or pressed on a plane)of the standard image plate for calibration under different gestures anddistances collected by the image sensing units, obtains feature cornerfrom the collected images and its corresponding coordinates in theimages, and outputs them to the system inherent internal-externalparameters obtaining module, which calculates and obtains the internalparameters of the image sensing units intended to be calibrated, andstores them into the non-volatile memory of the module, the calculationprinciple and method is the prior art in the field, for example usingZhengYou Zhang's method, here the concrete principle will not bedescribed again. The system inherent internal-external parametersobtaining module outputs the obtained internal parameters of two imagesensing units to the image sensing interface module of the interfaceunit, and, the image sensing interface module performs correction to theoptical aberration parameters of the images collected by the imagesensing units 1 and 2, according to the inherent internal parameters, toeliminate the optical aberration distortion brought by the property ofthe optical components of the image sensing units 1 and 2, and outputsthe corrected images to the image processing unit. After obtaining theinternal parameters of the image sensing units 1 and 2, and then toobtain the internal parameters and the system inherent externalparameters of the projecting unit, during the procedure of obtainingthese parameters, the graphics or images (such as black and white orcolorful chessboard) with obvious features are produced by the interfaceprocessing module or input by the outside of the system, additionally,the projecting plane is a standard image plate used in obtaining thesystem internal parameters of the image sensing units, or a blank flat.During the procedure of obtaining the parameters, the position andgesture of the projecting plane relative to the system is varied, theinterface processing module of the projected interface processing unitis for obtaining feature information (which of the projecting images atleast includes types of a basis feature unit, such as points, lines orother regular graphics or images, further includes the positioncoordinates of the basis feature unit in the projected human computerinterface images) of the images intended to be projected, the featureextracting module of the image processing unit obtains featureinformation (feature information which collected by the image sensingunits at least includes types of a basis feature unit, such as points,lines or other regular graphics or images, further includes the positioncoordinates of the basis feature unit in the images) of the images ofthe projection area collected by the image sensing units, these twokinds of feature information are sent to the feature extracting moduleof the image processing unit for performing the process of featurematching, and output to the system inherent internal-external parametersobtaining module, for calculating and obtaining the internal parametersand system inherent external parameters of the projecting unit, andstoring them in the nonvolatile memory of such module. Since the stepsof obtaining the system inherent internal and external parameters andobtaining feature, matching method, and calculating method are belong tothe prior art in this filed, the detailed will not be described here.The internal parameters of the projecting unit obtained by the systeminherent internal-external obtaining module are output to the projectinginterface module, which performs predistortion correction to theinterface image to be projected output by the projected interfaceprocessing unit based on such parameters, to eliminate the opticalaberration distortion brought by the property of the optical componentsof the projecting unit, and outputs the corrected interface image to theprojecting unit; the system inherent internal-external parametersobtaining module further outputs the calculated and obtained systeminherent external parameters to the feature extracting module.

2, Obtaining the System Variable External Parameters

On the basis of the obtained system inherent internal and externalparameters, the position and gesture relationship (that is the relativeposition and gesture between the projecting plane, and two image sensingunits and the projecting unit) of the projecting plane relative to thehuman computer interaction system is needed to be known before thesystem performs a projecting operation every time, i.e. to obtain systemvariable parameters. The aim for obtaining such parameters is to realizethe shape of the projected human computer interaction interface saw byusers will not be changed on the projecting plane. The detailedprocedure is similar to that of obtaining the system inherent externalparameters, the controlling unit controls the whole system in the stateof obtaining the system variable external parameters, the interfaceprocessing module of the projected interface processing unit producesgraphics or images with obvious feature (such as black and white orcolorful chessboard or structured light pattern) and outputs it to theprojecting interface module, or the graphics or images with obviousfeature is input from the outside of the system (that is, the graphicsor images with obvious feature is input by the interface inputtingmodule), the feature information of the images to be projected isobtained by the interface processing module of the projected interfaceprocessing unit, and output to the feature extracting module of theimage processing unit, at same time, the feature extracting module perse obtains feature information of the images of the correspondingprojection area collected respectively by the image sensing units 1 and2, and then, the feature extracting module of the image processing unitperforms the feature matching to the feature information of the imagesof such three units, and calculates and obtains the system variableexternal parameters (i.e. the relative position and gesture between theprojecting plane, and two image sensing units and the projecting unit)the matching process and calculating method is belong to the prior artin the field, here no detailed will be described. The feature extractingmodule outputs the relative position and gesture etc. parameters betweenthe projecting plane and the projecting unit of the system variableexternal parameters to the interface processing module of the projectedinterface processing unit, which performs shape predistortion correctionbased on the system variable external parameters, and outputs the resultto the projecting interface module; the feature extracting moduleoutputs the system inherent internal and external parameters and thesystem variable external parameters to the fingertip touch determiningmodule, as well as calculates and obtains homography matrixes (that is,conversion relationship of the coordinates between any points in twoplanes of the image sensing units and the plane of the projecting unit)between two planes of the two image sensing units (i.e. two planes ofthe two image sensor chips) and the plane of the projecting unit (i.e. aplane of the projecting chip), by utilizing the system inherent internaland external parameters and the system variable external parameters, andoutputs the homography matrixes to the fingertip touch determiningmodule.

3, Obtaining System Touch Operation

In order to realize projecting the human computer interaction interfaceon the planes in daily life, and performing bare hand operation on theprojecting plane similar to touch screen, one key point is to performdetermining that bare hands touched the projecting plane accurately,this exemplary embodiment will perform determining of the touchoperation with the principle of binocular disparity. After obtaining thesystem variable external parameters, that is obtaining the position andgesture relationship of the projecting plane relative to the humancomputer interaction system, the space position and gesture relationshipof the projecting plane relative to the projecting unit and two imagesensing unit is known, next step, the relationship between the hand andthe projecting plane is obtained by merely obtaining the space positionrelationship of the hand relative to two image sensing units or theprojecting unit, to determine whether the hand touched the projectingplane, and the touch position. During the procedure of obtaining thesystem touch operation according to this embodiment, the controllingunit controls the whole system in the state of obtaining the touchoperation, the fingertip obtaining module of the image processing unitreceives the images of the projection area collected by the imagesensing units 1 and 2, respectively extracts the fingertip from such twoimages based on the characteristics of hands, obtains the positioncoordinates of the fingertip respective in the planes of two imagesensing units, and outputs the result to the fingertip touch determiningmodule, which receives the system inherent internal and externalparameters output by the feature extracting module, and calculates theposition relationship of the fingertip relative to two image sensingunits with the principle of binocular disparity, and outputs theposition relationship to the fingertip touch determining module, whichcalculates the distance of the fingertip relative to the projectingplane utilizing the position and gesture parameters of the two imagesensing units relative to the projecting plane of the system variableexternal parameters, if the distance is in the range of one thresholdvalue, the touch operation is determined. The position of the touchpoint in the projected interface is also needed to be calculated if thetouch operation is occurred, preferred, the fingertip touch determiningmodule calculates the position of the touch point in this embodiment,the fingertip touch determining module calculates the coordinatesposition of the touch point in the originally projected interface, byusing the position coordinates (that is coordinates of the touch pointin the image sensing unit) of the fingertip in the image sensing unit 1or 2 output by the fingertip obtaining module, and the homography matrix(this is described in the procedure of obtaining the system variableexternal parameters above) between the image sensing unit 1 or 2 and theprojecting unit obtained from the feature extracting module, finally thefingertip touch determining module outputs the touch operation and theposition coordinates information of the corresponding touch point in thehuman computer interaction interface. Finally the fingertip touchdetermining module outputs the position coordinates of the touch pointin the human computer interaction interface.

Embodiment 2

Considering in the embodiment 1, the procedure for searching thefingertip with the property of the fingertip in the whole projectedhuman computer interaction interface and then determining the distancefrom the fingertip to the projecting plane is required, however, thewhole procedure is complex and time consuming. Therefore, suchembodiment will realize the coarse positioning of the finger byutilizing the structured light measuring system constituted by oneprojecting unit and one image sensing unit and the known obviouscharacteristics points, lines or other regular graphics or images etc.,in the human computer interaction interface. The core idea is to realizethe coarse positioning of the finger in the area of the projectedinterface by utilizing variation of the original feature informationresulted from that there is a finger present or no finger present in thehuman computer interaction interface projected on the projecting plane,to narrow a searching area of the fingertip, and lower the complexity ofthe fingertip searching, even to directly determine touching based onthe existing variation of the feature information, without searching thefingertip certainly, so that the complexity of the system may be loweredagain.

After coarse positioning of the finger, it comes into the state of finepositioning of the fingertip, it is performed that searching thefingertip in the area of the coarse positioning of the finger byutilizing two image sensing units, and finely measuring the distance ofthe fingertip from the projecting plane with the principle of binoculardisparity, the principle of the fine positioning of the fingertip issame as embodiment 1, here no detailed will be described.

Additionally, when the system performs the coarse positioning of thefinger, the fingertip touch determining of next step may be not happenedif the finger is too far from the projecting plane (for example the handis suspended over the projected interface) during coarse positioning,and then the touch is not occurred, therefore, computational cost issaved. Another effect of the embodiment is that one image sensing unitand one projecting unit is merely needed working during coarsepositioning of the finger, so that the power consumption for handlingother one image sensing unit to collect images is saved.

During the state of coarse positioning of the finger, fine positioningof the finger will be occurred, for example, when the basis feature unitin the projected interface is projected on the finger and the fingertipor near the fingertip, the position of the fingertip may be obtainedaccurately with the principle of structured light measuring, if thefingertip has touched the projecting plane, the accurate touchdetermining of the finger may be given, and meanwhile the positiontouched may be accurately calculated, and the result of touchdetermining may be output to the fingertip touch determining module.Considering the touch position is not needed calculating accurately, forexample if the finger clicks on a button with a large area, the resultsof touch determining and the area of the touched position may bedirectly given, and these results are output to the fingertip touchdetermining module when variation of the feature information is measuredin a certain range.

The structure of the human computer interaction system according to thisembodiment is shown in FIG. 2

The controlling unit is for controlling and coordinating the wholesystem into the state of the fingertip coarse positioning and thefingertip fine positioning; if the system is in the state of fingertipcoarse positioning, the controlling unit controls and selects anyone ofthe image sensing unit 1 and the image sensing unit 2 into a restingstate or to stop processing of images collected by the selected imagesensing unit, or, selects the nearest one between the image sensing unit1 and 2 distant from the projecting unit into a resting state or to stopthe feature extracting module processing images collected by theselected image sensing unit; if the system is in the state of fingertipfine positioning, the controlling unit controls the image sensing units1 and 2 in the normal working state, or controls to start the featureextracting module to perform normal processing to the images collectedby the image sensing units 1 and 2;

if the system is in the state of fingertip coarse positioning, thefeature extracting module calculates and obtains the homography matrixesbetween the plane of the projecting unit and the planes of the imagesensing units in the normal working state based on the system variableexternal parameters; receives the feature information 1 (which at leastincludes feature shape and the position coordinates of the basis featureunit) of the human computer interaction interface output by theinterface processing unit, or, receives human computer interactioninterface images intended to be projected output by the projectedinterface processing unit and extracts the feature information 1 of theimages through the feature extracting module; and then performscoordinate conversion of the feature information 1 with the homographymatrix, into a coordinate system of a plane of the image sensing unit inthe normal working state, and obtains a new corresponding featureinformation 2; and obtains feature information 3 (which at leastincludes feature shape and position coordinates of the basis featureunit) of the human computer interaction Interface on the projectingplane collected by the image sensing unit under the state of normalworking; the feature extracting module further performs matching to thefeature information 2 and the feature information 3, and then outputsthe matchable results (which at least include the matchable basisfeature unit pairs, and the position coordinates of the matchable basisfeature unit pairs respective in the planes of the image sensing unitsunder the normal working state) to the feature deformation measuringmodule; of course the feature extracting module may also performmatching to the feature information in a coordinate system of theprojecting unit, to obtain matchable basis feature unit pairs, andoutputs the matchable result to the feature deformation measuringmodule;

The human computer interaction system further includes the featuredeformation measuring module, which receives matchable result of featureinformation output by the feature extracting module, performs comparingto all matchable basis feature units, respectively calculates variationbetween the basis feature unit pairs, (the variation at least includesshape variation or area variation between the basis feature unit pairs,or offset of the position coordinates of feature points); The featuredeformation measuring module further calculates a distance of the fingerfrom the projecting plane at the point of feature deformation, andevaluates an area for the finger present in the image planes of theimage sensing units 1 and 2 based on the variation between the basisfeature unit pairs (that is the variation which is brought by the fingergot involved in the human computer interaction interface), 1) if thedistance from the finger to the projecting plane at the point of featuredeformation is smaller than a smaller threshold value, touch operationis directly determined; if the area of the touching is too large(evaluates that the possible area touched by the finger exceeds a singleclickable area of the interface), coordinates of the touching area isoutput to the fingertip obtaining module to obtain accurate touchingcoordinates, otherwise, coordinates of the corresponding touching areais directly output to the fingertip touch determining module, 2) if thedistance from the finger to the projecting plane at the point of featuredeformation exceeds a bigger threshold value, no processing is performedby following, 3) if the calculated distance from the finger to theprojecting plane at the point of feature deformation is between theaforesaid two threshold values, the coordinates of the finger evaluatedin the area in the image planes of the image sensing units 1 and 2 isoutput to the fingertip obtaining module;

The fingertip obtaining module is for receiving possible presenting areaof the fingertip in the human computer interaction interface images,which are collected by the two image sensing units 1 and 2, output bythe feature deformation measuring module, and searching the fingertip inthe corresponding area, and then outputting two-dimensional coordinatesof the fingertip on the image planes respectively collected by the imagesensing units 1 and 2 to the fingertip touch determining module;

The fingertip touch determining module is further for receivingfingertip touch information, which includes accurate coordinatesinformation of fingertip touch position and position information offingertip touching area, output by the feature deformation measuringmodule; further calculating and obtaining accurate position of thefingertip which touched the human computer interaction interfaceoriginally projected in accordance with the homography matrixes betweenthe two image sensing units or anyone of them, and the projecting unitwhen receiving touch operation output by the feature deformationmeasuring module and the accurate fingertip position coordinates of thefingertip in the planes of the image sensing units output by thefingertip obtaining module, and outputting the fingertip touchinformation, it should be understood that the two position coordinateswith two different values may be obtained because the calculation of theaccurate position coordinates uses the homography matrixes between twoimage sensing units and the projecting unit at the same time, therefore,the simplest method to obtain final position coordinates is to find theaverage of the two position coordinates values, to eliminate some errorbrought by the random noise.

This system is different from the embodiment 1 that the system mayintroduce a state of coarse positioning of the finger before finepositioning of the finger, the controlling unit controls the wholesystem into the state of obtaining touch operation, to perform coarsepositioning of the finger with one image sensing unit and one projectingunit, that is to evaluate the possible area of the fingertip in the areaof the projected interface, and then the controlling unit controls thewhole system into the state of fine positioning of the fingertip, andstarts processing on the images collected by the other one image sensingunit, searching the fingertip in the area obtained by coarse positioningof the finger, that is searching the fingertip in the appointed area ofthe images collected by two image sensing units, and determiningfingertip touch and obtaining touch position with the principle ofbinocular disparity. After the system is in the state of finepositioning of the finger, the procedure is same as the embodiment 1.Here, coarse positioning of the finger will be mainly detailed describedin the embodiment.

The controlling unit controls and coordinates the whole systemrespective into the states of coarse positioning of the finger and finepositioning of the fingertip. When the system is in the state of coarsepositioning of the finger, concerning the measurement precision, theembodiment will prefer the further one of the image sensing units 1 and2 with the relative distances between the image sensing units 1 and 2,and the projecting unit to work with the projecting unit together, inorder to describe conveniently, this selected image sensing unit iscalled by the image sensing unit S, while there is no processing to theimages collected by the other image sensing unit, or this image sensingunit may be suspended working into a resting state; When the system isin the state of coarse positioning of the finger, anyone of the imagesensing units 1 and 2 may be selected to work with the projecting unittogether, while there is no processing to the images output by the leftimage sensing unit which is not selected, or the left image sensing unitmay be suspended working into a resting state to save power consumption.After finishing coarse positioning of the finger, the controlling unitmay control the system into the state of fine positioning of the finger,and start normal working of the two image sensing units and performnormal processing on images of two way collected when determiningoperation of fine touch of the finger is still needed.

In order to obtain variation of the feature information of the humancomputer interaction interface on the projecting plane on which there ishand or no hand in the human computer interaction interface, firstly thefeature information of the human computer interaction interface on theprojecting plane without hands is to obtain, considering during theprojecting there is possibility that hands may always be in theprojection area of the human computer interaction interface, thereby itmay not insure that the image sensing units may collect the imageswithout hands, so that the embodiment will calculate the positioncoordinates and shape information, in the planes of the image sensingunit, of the feature information of the human computer interactioninterface projected on the projecting plane without hands, by utilizinghomography matrixes between the plane of the projecting unit and theplanes of the image sensing units, (or calculate the images on theplanes of the image sensing units mapped by the human computerinteraction interface images to be projected with homography matrixes,and extract the feature information of the images by the featureextracting module). Therefore, the feature extracting module convertsthe feature information of the human computer interaction interfacereceived from and output by the interface processing module (or receivethe human computer interaction interface images to be projected outputby the interface processing unit and extract the feature information ofthe images by the feature extracting module), into the planes of theimage sensing unit S, in accordance with the obtained homographymatrixes between the obtained plane of the projecting unit and the planeof the image sensing unit S (that is the corresponding coordinatesconversion relationship between the plane of the projecting unit and theplane of the image sensing unit S) under the condition the projectingplane is not changed relative to the position and gesture of the system,that is, evaluate the feature shape and the position coordinates (forshort, feature information projected and mapped in the image sensing) inthe plane of the image sensing unit S corresponding to the featureinformation of the human computer interaction interface collected by theimage sensing unit S, under the condition that during the currentprojecting plane, there is no hand present in the human computerinteraction interface projected on the projecting plane; the featureextracting module further extracts feature information (includingcharacteristic shape and position coordinates of the basis feature unit)of the images in the projection area collected by the image sensing unitS; and the feature extracting module further respectively matches thefeature information projected and mapped in the image sensing to that ofthe images in the projection area collected by the image sensing unit S,and outputs the matchable result (which at least includes the matchablebasis feature unit pairs and the position coordinates of the matchablebasis feature unit pairs in the plane of the image sensing unit S) tothe feature deformation measuring module;

The feature deformation measuring module receives the matchable resultbetween the feature information projected to be mapped in the plane ofthe image sensing unit output by the feature extracting module and thefeature information of the images of the projection area collected bythe image sensing unit S, respectively calculates shape variation, areavariation of their feature, or offset of the position coordinates of thefeature points and so on for each pair of matchable basis feature units.If the distance of the finger at the point of feature deformation fromthe projecting plane is smaller than one smaller threshold value, touchoperation is directly determined (the practical situation is at the timethe basis feature unit is projected on the finger, and projected on thefingertip or near the fingertip), here, if a possible touch area of thefinger is evaluated to exceed the range of a single clickable area ofthe interface of this area, then coordinates of the touch area is outputto the fingertip obtaining module to obtain more accurate touchcoordinates; if a possible touch area of the finger is evaluated to bein the range of a single clickable area of the interface of this area,then coordinates of the corresponding touch area is directly output tothe fingertip touch determining module; if the distance of the finger atthe point of feature deformation from the projecting plane is biggerthan one bigger threshold value, (the practical situation is at the timethe distance of the finger from the projecting plane is very far, i.e.the situation the fingertip touched the projecting plane may not happenwith this distance), next process is not needed, to save unnecessaryfingertip searching and touch operation; if the distance of the fingerat the point of feature deformation from the projecting plane is betweenthe aforesaid two threshold values, coordinates of the fingertip of thepossible area evaluated in the image planes of the image sensing units 1and 2 is output to the fingertip obtaining module. As shown in FIG. 3,FIG. 3 (a) illustrates the case where the human computer interactioninterface is projected on the projecting plane, four obviouscharacteristic streak lines and one obvious characteristic point areincluded, FIG. 3 (b) illustrates the case where there is hands or touchelement operated by the user on the projection area of the projectingplane, it is found that because the hand or other touch elementpresents, the basis feature unit on right-hand i.e. stripe onright-hand, performs stripe deformation in part, it is easily known thatthe deformation of stripe completely contains altitude information ofthe hand involved in the projection area at the deformed stripe from theprojecting plane (calculating method of the altitude is belong to theprior art in this filed, here it will not be detailed described),therefore, according to the principle of the structured light measuringsystem constituted by one image sensing unit and one projecting unit,the feature deformation measuring module may calculate the altitude ofthe hand involved at the deformed stripe from the projecting planethrough measuring this deformation value or offset, and determinewhether the hand presents in the area of the projected interface basedon the variation of the altitude information, for example, luminance andcolors of the projection area of the projecting plane is clearlydifferent from that of non-projection area, so four boundary lines maybe formed. Four lines are shown in FIG. 3 (i.e. four outermost lines ofthe projected interface), whether there is hand or other touch elementpresenting in the projected human computer interaction interface may bedetermined through determining whether anyone of four lines deforms, ifthere is no deformation, the following finger searching or touchoperation is not needed, to save power consumption; if the basis featureunit is projected on the fingertip or near the fingertip (the basis unitis still projected on the finger), as shown in FIG. 3 (c) the featurepoint is on the fingertip, here, the altitude of the fingertip from theprojecting plane is accurately (the basis feature unit is on thefingertip) or substantially accurately (the basis feature unit is nearthe fingertip) measured with one projecting unit and one image sensingunit, if the altitude is belong to one of threshold values, the touchoperation is directly determined, and the touch operation information isoutput to the fingertip touch determining module. Additionally, if thereis too many basis feature units in the projected interface, as shown inFIG. 4, for example, the obvious boundary line may present betweeninterface windows, where there is multiple interface windows in theprojected human computer interaction interface, here, the basis featureunits are stripes, the fingertip may be evaluated and forecasted topresent in the area of the projected interface by determining theposition of line deformation and its deformation variation when there istoo many feature stripe present in the projected human computerinteraction interface, as shown in FIG. 4 (a), since characteristicstripe a in the window A area is deformed in part, and the other fourboundary lines are not deformed, the fingertip located in the area isdetermined area A, while two boundary lines a and b are deformed in thewindow B area, considering the case where there are multiple fingerstouching the projected interface, firstly, it is determined whetherdeformation of the line b in part is smaller than one threshold value(that is determining whether the altitude of the finger at the deformedline from the projecting plane is smaller than one threshold value), ifit is, then the finger is also determined to be present in the window Barea, otherwise, it is determined whether deformation in part exceedsone threshold value (that is determining whether the altitude of thefinger at the deformed line from the projecting plane exceeds onethreshold value), here, there is no possibility the fingertip touchedthe projecting plane of the window B area, then whether the fingertip isin the area B is not concerned, to be simplified, under this situation,the result that there is no fingertip may be given. If the deformationin part is between the aforesaid two threshold values, then it isdetermined that whether the fingertips at the deformation in part of thetwo characteristic lines a and b have connected finger boundary, ifhaving, it is determined that there is no fingertip in the window B area(as shown in FIG. 4 (a)), otherwise, there is fingertip in the window Barea, finally, the range of the fingertip presenting area is output tothe fingertip obtaining module, to finely search the fingertip. Whenthere is more basis feature units distributed in the projected humancomputer interaction interface, as shown in FIG. 4 (b), the samedetermining procedure and method is performed to further narrow thefingertip searching scope to A4 area. When there are more basis featureunits presenting in the human computer interaction interface projectedon the projecting plane, that is the interval of the positions of thebasis feature units is much smaller, as shown in FIG. 5, there arecharacteristic lines with small intervals in the interface, such askeyboard in the interface (here, it is merely determined which key-pressarea is touched by the finger, the precision of the touch position isnot needed), here, since the interval between lines is relative small,during the touch operation between every two lines, the distance of thefinger at the deformation from the projecting plane is measured with theprinciple of structured light, if the measured distance is smaller thanone threshold value, it is directly determined whether there isfingertip touch, and the touched area of the fingertip (an area occupiedby one key-press) is output, and the result is directly output to thefingertip touch determining module, therefore, the computationcomplexity of the system may be greatly lowered under this case.According to the above description, the feature deformation measuringmodule outputs the possible area of the fingertip in the images,collected by the two image sensing units, of the human computerinteraction interface to the fingertip obtaining module under the stateof coarse positioning of the finger; and further outputs fingertip touchinformation to the fingertip touch determining module, which includesaccurate coordinates information of the position of the fingertip touch,and the position information of the area of the fingertip touch.

The fingertip obtaining module is for receiving possible presenting areaof the fingertip in the human computer interaction interface images,which are collected by the two image sensing units, output by thefeature deformation measuring module, and searching the fingertip insuch area, then outputting two-dimensional coordinates of the fingertiprespectively on the plane of the images collected by the image sensingunits 1 and 2 to the fingertip touch determining module;

The fingertip touch determining module is further for receivingfingertip touch information, which includes accurate coordinatesinformation of fingertip touch position and position information offingertip touching area, output by the feature deformation measuringmodule; further calculating and obtaining accurate position of thefingertip which touched the human computer interaction interfaceoriginally projected in accordance with the homography matrixes betweenthe planes of the image sensing units and the projecting unit whenreceiving fingertip touch operation output directly by the featuredeformation measuring module and the accurate fingertip positioncoordinates of the fingertip in the planes of the image sensing unitsoutput by the fingertip obtaining module, and outputting the fingertiptouch information.

Embodiment 3

Considering the different projecting planes will have different texturecolors, the system may finish obtaining the projecting colorpredistortion parameters before projecting the human computerinteraction interface. This embodiment is supplement on the basis ofembodiments 1 and 2, here the consistent contents will not be described.The different parts, that is obtaining the projecting colorpredistortion parameters and color predistortion correcting, will bemainly described, the function of these two parts will only implementedin the feature extracting module and the interface processing module inthis embodiment. Considering there are different colors and reflectionproperties on the different projecting planes, the system further needsscanning the colors and reflection properties of the projecting planesbefore the user performs projecting operation, that is the interfaceprocessing module of the projected interface processing unit producesimages with different intensity and different colors (which may be theimages with different intensity and colors input by the outside of thesystem and received through the interface inputting module), and sendsto the projecting unit to be projected on the projecting plane, andmeanwhile sends to the feature extracting module of the image processingunit, and then the image sensing units 1 and 2 collects the projectedimages on the corresponding projecting plane, the feature extractingmodule performs image comparison according to the images collected bythe image sensing units and the projected images produced by theinterface processing module (or images input by the outside), to obtaindifference analysis between the images intended to be projected by thesystem and the projected images on the projecting plane, for obtainingoptical modulation characteristics of the projecting plane, which isdenoted as function f (or a look up table), so the projecting colorpredistortion processing function is f⁻¹ (or a look up table). Thefeature extracting module outputs the obtained projecting colorpredistortion processing function f⁻¹ (or a look up table) to theprojecting interface processing unit, supposing the color of the imagesintended to be projected is Color, that is when the color of the humancomputer interface images received from the interface inputting moduleby the interface processing module is Color, the interface processingmodule performs preprocessing f⁻¹ (Color) to the images, the projectedcolor of which is Color, according to the projecting color predistortionprocessing function, so that the image projected on the projecting planeis f (f⁻¹ (Color))=Color, to realize projecting the human computerinteraction interface with accurate colors.

Embodiment 4

This embodiment 4 is different from embodiments 1, 2 and 3 that thehuman computer interaction interface information input to the interfaceinputting module may include feature information of the human computerinteraction interface to be projected which includes basis feature unittypes (at least includes points, lines, geometrical shape and so on),and the position coordinates of the basis feature unit, except includeimages of the human computer interaction interface, for example theimage to be projected is chessboard, so the human computer interactioninterface information may include types of the basis feature unit (atleast include points, lines, geometrical shapes and so on), and theposition information of the feature corner of the chessboard (i.e.position coordinates information of the basis feature unit) and so on;for another example, the image to be projected is the keyboard as shownin FIG. 5, so the human computer interaction interface information mayinclude the position coordinates information of every stripe in theprojected interface. The benefits is calculating feature information inadvance with the known interface to be projected, thereby thecomputation complexity of the interface processing module of theprojected interface processing unit may be greatly lowered, and powercost brought by the feature extracting to the projected images etc., mayalso be lowered.

The human computer interaction interface information input to theinterface inputting module further includes ranges of every clickablearea in the human computer interaction interface, such as area ofbuttons, icons, menu etc.;

The interface processing module further extracts the ranges of everyclickable area in the human computer interaction interface and outputsit to the feature deformation measuring module;

The feature deformation measuring module further receives the ranges ofevery clickable area in the human computer interaction interface fromthe interface processing module, when the system is in the state offingertip coarse positioning, determines whether evaluated positionrange of the finger of the coarse positioning is in the range inaccordance with the ranges of every clickable area, if it is, thendirectly outputs touch information to the fingertip touch determiningmodule; if not, then outputs evaluated position ranges of the finger ofthe coarse positioning to the fingertip obtaining module, and sendsfingertip touch signals to the fingertip touch determining module; Theinterface processing module further directly extracts featureinformation data of the interface from the human computer interactioninformation.

Embodiment 5

Considering the projecting plane is not fixed, there is rotation ortranslation of the projecting plane relative to the human computerinteraction system, here, under the controlling of the controlling unit,the feature extracting module is started to obtain the system variableexternal parameters at regular time, that is to obtain the relativeposition and gesture information of the projecting plane relative to thehuman computer interaction system, when the relative position andgesture is found changed, the changed system variable externalparameters is obtained again, and output such system variable externalparameters to the interface processing module again, the interfaceprocessing module updates these parameters, and performs the projectingshape predistortion processing based on such new parameters, to realizetracking the projecting plane in real time.

It should be understood that all embodiments of the present inventionhave described the case where the hand touched the projecting plane,here, the other touch element can replace the hand, but the principle issame, so it will not be described detailed again.

As shown in FIG. 6, it is a flowchart of the working method of the humancomputer interaction system, which comprises the steps of:

Step S1, initially calibrating the human computer interaction system, toobtain system inherent internal-external parameters, which include theinternal parameters of the projecting unit, image sensing units 1 and 2,and the external parameters such as relative space position and gesturerelationship among these three units; and storing the system inherentinternal and external parameters in the nonvolatile memory of the systeminherent internal-external parameters obtaining module;

Step S2, obtaining system variable external parameters, that is therelative space position and gesture relationship between the projectingplane and the human computer interaction system which includes therelative space position and gesture relationship respective between theimage sensing units 1 and 2, the projecting unit and the projectingplane, and calculating homography matrixes respectively between theplane of the projecting unit, and the planes of the image sensing units1 and 2 based on the system variable external parameters and the systeminherent external parameters;

Step S3, finely obtaining, position coordinates of a fingertip on theplanes of images collected by the image sensing units 1 and 2, andcalculating a distance from the fingertip relative to the projectingplane, and searching the fingertip, by the feature extracting unit, fromimages received from and collected by the image sensing units 1 and 2,and outputting the obtained position coordinates of the fingertip in thetwo planes of the image sensing units 1 and 2 to the fingertip touchdetermining module;

Step S4, determining whether the touch operation is occurred:calculating, by the fingertip touch determining module, a positionrelationship of the fingertip relative to the image sensing units,utilizing the principle of binocular disparity, in accordance with thereceived position coordinates of the fingertip respective in the planesof the image sensing units 1 and 2, and calculating the relativeposition and gesture between the image sensing units and the projectingplane based on the system variable external parameters, finallycalculating and obtaining the distance of the fingertip from theprojecting plane, if the distance is smaller than a threshold value,determining a touch operating is occurred, and then turning to step S5;Otherwise, returning to step S3;

Step S5, outputting touch information: calculating and obtainingaccurate position coordinates of the touching points in the coordinatessystem of the human computer interaction interface, in accordance withthe coordinates of the touch points in the planes of the image sensingunits, combining with homography matrixes respectively between theprojecting unit, and the first image sensing unit and/or the secondimage sensing unit obtained by step S2, and then outputting them, andreturning to step S3. It should be understood that if the accurate touchposition coordinates is calculated with homography matrixes between thetwo image sensing units and the projecting unit at the same time, twoposition coordinates with a little different value may be obtained, thesimplest method is to found the average of two position coordinates asfinal position coordinates to eliminate some error brought by the randomnoise.

Step S3 in the above mentioned method may further perform coarsepositioning to the possible area of the fingertip, by utilizing theexisting obvious feature information of the human computer interactioninterface and structure light measuring system constituted by one imagesensing unit and one projecting unit, and then finely search thefingertip in the images collected by the image sensing units 1 and 2 tofurther narrow the area of searching the fingertip.

The step S31: the step of coarse positioning of the fingertip. Thecontrolling unit controls the human computer interaction system into thestate of fingertip coarse positioning, selects anyone image sensing unitinto the state of resting from the first image sensing unit and thesecond image sensing unit, or stops all modules of the image processingunit processing on the images collected by the selected image sensingunit; or selects the nearest one from the first image sensing unit andthe second image sensing unit distant from the projecting unit into theresting state, or stops all modules of the image processing unitprocessing on the images collected by the selected image sensing unit,to further obtain, by the feature extracting module, some basis featureunit pairs matched by a first basis feature unit and a second basisfeature unit, to further calculate and obtain variation between thebasis feature unit pairs by utilizing the feature deformation measuringmodule, for performing a coarse positioning of the possible area thatthe fingertip presents in the images collected by the image sensing unitselected to work.

The interface processing module obtains obvious feature of the interfaceimages from the input human computer interaction interface information,and converts the extracted feature information, (which is briefly calledthe feature information of the original interface), into the image planeof the image sensing unit with the homography matrixes between the planeof the projecting unit and the plane of the image sensing units,extracts the feature information (which is briefly called interfacefeature information of the projecting plane) from the collected images(which include the case where the finger of the user is operated in theprojection area of the human computer interaction interface) of thehuman computer interaction interface collected by and projected on theprojecting plane by utilizing the feature extracting unit, and outputsthe feature information of the original interface and that of theinterface of the projecting plane to the feature deformation measuringmodule, which compares variation of such two feature information (whichat least includes shape variation, area variation or offset of theposition coordinates of the feature points etc.,) for coarse positioningthe position of the possible area of the finger present in the imagescollected by the image sensing unit, for different practical situations:1, if the basis feature unit is projected on or near the fingertip (thebasis feature unit is still projected on the finger), as shown in FIG. 3(c) the feature point is on the fingertip, here, the distance of thefinger from the projecting plane may be measured accurately (the basisfeature unit is projected on the fingertip) or roughly accurately (thebasis feature unit is projected near the fingertip) by utilizing oneprojecting unit and one image sensing unit, if the distance is in therange of one threshold value, the touch operation is directlydetermined, and the position information of the fingertip in the imagesis accurately calculated, then turning to step S5, otherwise, returningto step S31 to operate again; 2, when the human computer interactioninterface projected on the projecting plane has dense basis featureunits, that is the interval of the position of the basis feature unitsis much small, and the computation precision of the position of thetouch points is not high, as shown in FIG. 5, there are feature stripeswith small intervals in the interface, for example, the keyboard part ofthe interface, and here, it is only to be known which keystroke thefinger touched, and the computation precision to the touch position isnot needed accurate, when the fingertip performs touch operation betweenevery two stripes, the distance of the fingertip from the projectingplane at the deformation are measured with the principle of structuredlight, if the distance is smaller than a threshold value, the fingertiptouch operation is directly determined, and the area touched by thefingertip (which is in the range of area occupied by the keystroke, i.e.single clickable area) is merely obtained, and the touch operation andthe position information of the area touched by the fingertip is outputto the touch determining module, directly turning to step S5, otherwisereturning to step S31, to start again, therefore, the computationcomplexity of the system may be greatly lowered in this situation; 3, ifthere are relative non-dense basis feature units in the projectedinterface, as shown in FIG. 4, for example, when there are multipleinterface windows in the projected human computer interaction interface,the obvious boundary lines will be in the interface windows, thepossible area where the fingertip presents may be evaluated throughdetermining the variation of the basis feature units (i.e. the positionand size of the stripes deformation) smaller than a threshold value, butdetermining whether the fingertip has performed touch operation may notbe given, and, the procedure proceeds to step S32, if the variation ofthe basis feature unit is determined bigger than a threshold value, andthe distance of the finger from the projecting plane is determined muchfar, so there is no possibility that touch operation is occurred, thenreturning to step 31 to restart;

Step S32: finely searching a fingertip, in accordance with the coarsepositioning area of the finger obtained by step S31, this step includessearching, by the fingertip obtaining module, the fingertip in theappointed area in the images received from and collected by the imagesensing units 1 and 2, and outputting the position coordinates of thefingertip obtained in the image planes of the two image sensing units tothe fingertip touch determining module, then turning to step S4.

Considering different projecting planes have different texture colors,the system may further finish obtaining the projecting colorpredistortion parameters, before projecting the human computerinteraction interface,

Therefore, the step S2 further comprises scanning and obtaining aprojecting color predistortion parameter of the projecting plane andperforming color predistortion correction. Concerning the projectingplane is not fixed, the rotation or translation of the projecting planerelative to the human computer interaction system will appear, so thatstep S2 further comprises starting step S2 at regular time per se, thatis the feature extracting module is started to obtain the systemvariable external parameters at regular time (that is to obtain therelative position and gesture information of the projecting planerelative to the system), when the relative position and gesture is foundchanged, the varied parameters are output to the interface processingmodule again, which updates these parameters, and performs projectingshape predistortion processing based on such new parameters, to realizetracking the projecting plane in real time.

Other aspects and character of the present invention is apparent to theskilled in this field by the description of the embodiments of theinvention accompanying with the drawings.

The above mentioned is only the preferred embodiments of the presentinvention, it should be understood, the skilled in this technical fieldmay further make amendment and replacement without beyond the scope ofthe technical principle of the invention, these amendment andreplacement should be considered in the scope of the present invention.

What is claimed is:
 1. A system for human computer interaction, whereinthe system comprises: a projecting unit, a first image sensing unit, asecond image sensing unit, an interface unit, an image processing unit,a projected interface processing unit, and a controlling unit, in which,the projecting unit is for receiving human computer interactioninterface information data output by the interface unit, and projectingand outputting the data to a projecting plane; the first image sensingunit and the second image sensing unit is for collecting projected imageinformation on an area of the projecting plane, and outputting to theimage processing unit through the interface unit; the interface unit iscoupled to the projecting unit, the first image sensing unit, the secondimage sensing unit, the image processing unit, the projected interfaceprocessing unit and the controlling unit, for synchronously controllingthe first image sensing unit, the second image sensing unit and theprojecting unit; the image processing unit is for receiving image datacollected by the first image sensing unit and the second image sensingunit; receiving a human computer interaction interface image output bythe projected interface processing unit, and extracting featureinformation in the human computer interaction interface image;calculating and obtaining system inherent internal parameters whichcomprise aberration distortion parameters, focus, pixel size and centerpoint position on an optical lens of the first image sensing unit, thesecond image sensing unit and the projecting unit; storing the systeminherent internal parameters on a non-volatile memory, and outputtingthe system inherent internal parameters to the interface unit;extracting corresponding feature information for the image data, andmatching it with the feature information output by the projectedinterface processing unit; calculating and obtaining system inherentexternal parameters which comprise position and gesture relationshipamong the first image sensing unit, the second image sensing unit andthe projecting unit; and calculating and obtaining system variableexternal parameters which comprise position and gesture relationshipbetween the human computer interaction system and the projecting plane,and outputting position and gesture relationship parameters between theprojecting plane and the projecting unit in the obtained system variableexternal parameters to the projected interface processing unit; theimage processing unit is further for processing images of a projectionarea, which are collected by the first image sensing unit and the secondimage sensing unit, obtained through the interface unit; searching afingertip or other touch element according to a predetermined fingertipfeature, and obtaining a distance between the fingertip or other touchelement and the projecting plane based on a principle of binoculardisparity, if the distance is in the range of a predetermined firstthreshold value; determining a touch operation has occurred andoutputting touch information; the projected interface processing unit isfor receiving human computer interaction information intended to beprojected, extracting feature information from the human computerinterface interaction information, and outputting the featureinformation to the image processing unit; the projected interfaceprocessing unit is further for receiving the position and gestureparameter between the projecting plane and the projecting unit in thesystem variable external parameters output by the image processing unit,performing shape predistortion correction processing to interface imagesin the input human computer interaction interface information based onthe parameter and then outputting to the interface unit; the controllingunit is for controlling all units of the system, and coordinatelycontrolling work of all units of the system; the controlling unit isfurther for controlling the human computer interaction system in thestatuses of obtaining the system inherent internal or externalparameters, obtaining the system variable external parameters, andobtaining a fingertip and touch operation.
 2. The system for humancomputer interaction of claim 1, wherein, the interface unit includes animage sensing interface module and a projecting interface module,wherein, the image sensing interface module is for receiving the systeminherent internal parameters output by the image processing unit;receiving image data from the first image sensing unit and the secondimage sensing unit, and outputting to the image processing unit afterperforming optical distortion correction of the image data; theprojecting interface module is for receiving the projected image dataoutput by the projected interface processing unit, performing opticalaberration predistortion correction processing of the projectedinterface images output by the projected interface processing unit basedon the system inherent internal parameters, and outputting the correctedprojected interface images to the projecting unit.
 3. The system forhuman computer interaction of claim 2, wherein, the image processingunit includes a system inherent internal-external parameters obtainingmodule, a feature extracting module, a fingertip obtaining module, and afingertip touch determining module, wherein, the system inherentinternal-external parameters obtaining module is for receiving thesystem inherent internal or external parameters output by the featureextracting module, and storing the parameters in a non-volatile memory;outputting the system inherent internal parameters to the projectinginterface module and the image sensing interface module; and outputtingthe system inherent internal or external parameters stored in thenon-volatile memory to the feature extracting module; the featureextracting module is for receiving feature information of human computerinteraction interface output by the projected interface processing unit,under the state where the human computer interaction system is obtainingthe system inherent internal-external parameters and receiving imagesoutput by the image sensing interface module in accordance with thefeature information, extracting the corresponding feature information ofthree from images of the projection area collected by the first imagesensing unit and the second image sensing unit, performing matching ofthe feature information, calculating and obtaining the system inherentinternal-external parameters, and outputting the system inherentinternal-external parameters to the system inherent internal-externalparameters obtaining module; the feature extracting module is forreceiving feature information of human computer interaction interfaceoutput by the projected interface processing unit, under the state wherethe human computer interaction system is obtaining the system variableexternal parameters, receiving images output by the image sensinginterface module in accordance with the feature information, extractingthe corresponding feature information of three from images of theprojection area collected by the first image sensing unit and the secondimage sensing unit, performing matching of the feature information,calculating and obtaining the system variable external parameters,receiving the system inherent internal-external parameters output by thesystem inherent internal-external parameters module, calculating andobtaining homography matrixes between the first image sensing unit andthe projecting unit and between the second image sensing unit and theprojecting unit, and outputting the matrixes to the fingertip touchdetermining module; the fingertip obtaining module is for receivingimage data output by the image sensing interface module, under the statewhere the human computer interaction system is obtaining a fingertip andtouch operation, and processing the image data to obtain positioncoordinates information for the fingertip of the finger or front end oftouch elements in image planes of the two image sensing unitsrespectively, and outputting the position coordinates information to thefingertip touch determining module; the fingertip touch determiningmodule is for receiving the position coordinates information and thehomography matrixes, under the state of obtaining a fingertip and touchoperation; calculating a distance between the fingertip and the firstimage sensing unit and a distance between the fingertip and the secondimage sensing with the principle of binocular disparity, and,calculating and obtaining a distance between the projecting plane andthe fingertip, or the front end of the other touch element, by means ofposition and gesture relationship between the first image sending unitand the projecting plane and between the second image sensing unit andthe projecting plane.
 4. The system for human computer interaction ofclaim 3, wherein, if the distance between the projecting plane and thefingertip, or between the projecting plane and the front end of theother touch element is smaller than the predetermined first thresholdvalue, then touch operation is determined; if the touch operation isoccurred, the fingertip touch determining module is further forcalculating and obtaining coordinates position of the touch points inthe human computer interaction interface originally projected inaccordance with the homography matrixes, and outputting the fingertiptouch information.
 5. The system for human computer interaction of claim2, wherein, the projected interface processing unit includes aninterface processing module and an interface inputting module; wherein,the interface inputting module is for receiving human computerinteraction interface information data input by the outside, andoutputting to the interface processing module; the interface processingmodule is for receiving the human computer interaction interfaceinformation data output by the interface inputting module, receiving therelative position and gesture parameters between the projecting planeand the projecting unit from the system variable external parameters,and performing image geometric predistortion processing on the humancomputer interaction interface images, and outputting it to theinterface unit; the interface processing module is further forextracting feature information of the human computer interactioninterface from the input human computer interaction interfaceinformation, and outputting the feature information of the humancomputer interaction interface to the image processing unit.
 6. Thesystem for human computer interaction of claim 5, further comprising afeature deformation measuring module, wherein the controlling unit isfor coordinating the human computer interaction system into the state offingertip coarse positioning or fingertip fine positioning, if the humancomputer interaction system is in the state of finger coarsepositioning, the controlling unit controls and selects anyone of thefirst image sensing unit and the second image sensing unit into aresting state, or stops all modules in the image processing unitprocessing images collected by the selected image sensing unit, or,selects the nearest one between the first image sensing unit and thesecond image sensing unit distance from the projecting unit into aresting state or stops all modules in the image processing unitprocessing images collected by the selected image sensing unit; if thehuman computer interaction system is in the state of fingertip finepositioning, the controlling unit controls both the first image sensingunit and the second image sensing unit into a normal working state, orcontrols to start processing of images collected by the first imagesensing unit and the second image sensing unit, through all modules inthe image processing unit; if the human computer interaction system isin the state of fingertip coarse positioning, the feature extractingmodule calculates and obtains the homography matrix between theprojecting unit and the image sensing unit in the working state based onthe system variable external parameters, receives the featureinformation of the human computer interaction interface output by theinterface processing module, or receives human computer interactioninterface image intended to be projected output by the projectedinterface processing unit, and extracts the feature information of thehuman computer interaction interface images through the featureextracting module; performs coordinate conversion to the featureinformation with the homography matrix, into a coordinate system of animage plane of the image sensing unit in the working state, and obtainscorresponding feature information combined by a plurality of first basisfeature units; obtains feature information combined by a plurality ofsecond basis feature units in the human computer interaction interfaceon the projecting plane collected by the image sensing unit in theworking state, the feature extracting module performs matching to thefeature information combined by the plurality of first basis featureunits and that combined by the plurality of second basis feature unitsin a coordinate system of an image plane of the image sensing unit or ina coordinate system of an image plane of the projecting unit, to obtainmatchable basis feature unit pairs, and outputs the matchable result tothe feature deformation measuring module; the feature deformationmeasuring module receives matchable result of the feature informationoutput by the feature extracting module, performs comparing to allmatchable basis feature units, respectively calculates variation betweenthe basis feature unit pairs, which at least includes shape variation orarea variation between the basis feature unit pairs, or offset of theposition coordinates of feature points; calculates a distance from thefinger to the projecting plane at the point of deformation of the secondbasis feature unit based on the variation between the basis feature unitpairs, and evaluates the area for the finger present in the image planeof the first image sensing unit and the second image sensing unit, ifthe distance from the finger to the projecting plane at the point ofdeformation of the second basis feature unit is smaller than thepredetermined first threshold value, touch operation is directlyoccurred; if the area of the touching is too large, outputs coordinatesof the possible touching area to the fingertip obtaining module toobtain accurate touching coordinates, otherwise, directly outputscoordinates of the corresponding touching area to the fingertip touchdetermining module, if the distance from the finger to the projectingplane at the point of deformation of the second basis feature unitexceeds the predetermined second threshold value, no processing isperformed by following, if the calculated distance from the finger tothe projecting plane at the point of deformation is between the firstand the second threshold value, outputs the coordinates of the fingerevaluated in the area in the image plane of the first image sensing unitand the second image sensing unit; wherein the predetermined firstthreshold value is smaller than the predetermined second thresholdvalue; the fingertip obtaining module is for receiving possiblepresenting area of the fingertip in the human computer interactioninterface images, which are collected by the first image sensing unitand the second image sensing unit, output by the feature deformationmeasuring module, and respectively searching the fingertip only in thetwo corresponding area, then outputting two-dimensional coordinates ofthe fingertip respectively on the image planes collected by the firstimage sensing unit and the second image sensing unit to the fingertiptouch determining module; the fingertip touch determining module isfurther for receiving fingertip touch information, which includesaccurate coordinates information of fingertip touch position andposition information of fingertip touching area, output by the featuredeformation measuring module; further calculating a distance between thefingertip and the projecting plane with the principle of the binoculardisparity when receiving touch operation output by the featuredeformation measuring module and the accurate fingertip positioncoordinates of the image planes of the image sensing units output by thefingertip obtaining module, if the distance is smaller than thepredetermined first threshold value, calculating and obtaining accurateposition which the finger touched the human computer interactioninterface originally projected in accordance with the homographymatrixes between the projecting unit and the first image sensing unitand/or the second image sensing unit, and outputting the fingertip touchinformation.
 7. The system for human computer interaction of 6, whereinthe projected interface processing unit receives graphics or imageinformation of the human computer interaction interface input by theoutside, or further receives feature information extracted from thehuman computer interaction interface, which includes types of the basisfeature unit having points, lines, planes, letters or regular graphics,or images, and position information of the basis feature unit in thehuman computer interface images; the interface processing module furtherdirectly extracts feature information data in the interface from thehuman computer interaction information; the human computer interactioninterface information of the interface inputting module further includesranges of every clickable area in the human computer interactioninterface; the interface processing module further extracts the rangesof every clickable area in the human computer interaction interface andoutputs it to the feature deformation measuring module; the featuredeformation measuring module further receives the ranges of everyclickable area in the human computer interaction interface from theinterface processing module, when the system is in the state offingertip coarse positioning, determines whether evaluated positionrange of the finger of the coarse positioning is in the range inaccordance with the ranges of every clickable area, if it is, thendirectly outputs touch information to the fingertip touch determiningmodule; if not, then outputs evaluated position ranges of the finger ofthe coarse positioning to the fingertip obtaining module, and sendsfingertip touch signals to the fingertip touch determining module. 8.The system for human computer interaction of claim 7, wherein theinterface processing module is further for producing images withdifferent intensity or colors, to output them to the projectinginterface module and the feature extracting module for obtainingprojecting color predistortion processing function; the interfaceprocessing module is further receiving images with different intensityor colors from the outside of the system, to output them to theprojecting interface module and the feature extracting module forobtaining projecting color predistortion processing function; theinterface processing module is for receiving projecting colorpredistortion processing function f⁻¹ or a lookup table output by thefeature extracting module, the interface processing module is furtherfor performing color aberration pre-correction processing to the inputhuman computer interaction interface, to realize colors of the humancomputer interface projected on the projecting planes do not distort;the feature extracting module is further for receiving images withdifferent intensity or colors output from the interface processingmodule, further for receiving corresponding images with differentintensity or colors collected by the first image sensing unit and thesecond image sensing unit, and performing the images comparingprocessing, to obtain difference analysis between images intended to beprojected by the system and projected images of the projecting planes,to obtain optical modulation property of the projecting planes, andobtain projecting color predistortion processing function f⁻¹ or alookup table in accordance with the optical modulation property, thefeature extracting module outputs the obtained projecting colorpredistortion processing function f⁻¹ or the lookup table to theprojecting interface processing module.
 9. The system for human computerinteraction of claim 8, wherein the controlling unit is further forstarting the feature extracting module to obtain the system variableexternal parameters at regular time, that is obtain the relativeposition and gesture information of the projecting plane relative to thesystem; the feature extracting module is further for comparing thecurrent obtained system variable external parameters to the earlierobtained system variable external parameters, if the current obtainedsystem variable external parameters is different from the earlierobtained system variable external parameters, outputting the variedsystem variable external parameters to the interface processing moduleagain, and further for calculating the homography matrix between theprojecting unit, and the first and second image sensing unit, andoutputting to the fingertip touch determining module.
 10. A method forhuman computer interaction performed by the human computer interactionsystem as claimed in claim 1, wherein the method comprises: S1,initially calibrating the human computer interaction system, to obtainsystem inherent internal and external parameters; S2, obtaining systemvariable external parameters, and calculating homography matrixesrespectively between the projecting unit, and the first and second imagesensing unit based on the system variable external parameters and thesystem inherent external parameters; S3, finely obtaining, by afingertip obtaining module, position coordinates of a fingertip onplanes of images collected by the first image sensing unit and thesecond image sensing unit and calculating a distance from the fingertiprelative to a projecting plane, and searching the fingertip, by thefeature extracting unit, from images received from and collected by thefirst image sensing unit and the second image sensing unit, andoutputting the obtained position coordinates of the fingertip in theimage planes of the first image sensing unit and the second imagesensing unit to the fingertip touch determining module; S4, calculating,by a fingertip touch determining module, a position relationship of thefingertip relative to the image sensing units, by utilizing theprinciple of binocular disparity, in accordance with the receivedposition coordinates of the fingertip respective in the planes of thefirst image sensing unit and the second image sensing unit, andcalculating the relative position and gesture between the image sensingunits and the projecting plane based on the system variable externalparameters, and calculating and obtaining the distance of the fingertipfrom the projecting plane, if a distance between the fingertip and theprojecting plane is smaller than a predetermined first threshold value,determining touch operation has occurred, and then turning to step S5;otherwise, returning to step S3; S5, outputting touch information, andcalculating and obtaining accurate position coordinates of the touchingpoints in the coordinates system of the human computer interactioninterface, in accordance with the coordinates of the touch points in theimage planes of the first image sensing unit and/or the second imagesensing unit combined with homography matrixes respectively between theprojecting unit, and the first image sensing unit and/or the secondimage sensing unit, and then outputting them.
 11. The method of claim10, wherein the step S3 further comprises: S31, controlling, by thecontrolling unit, the human computer interaction system into the stateof fingertip coarse positioning, selecting anyone image sensing unitinto the state of resting from the first image sensing unit and thesecond image sensing unit, or stopping all modules of the imageprocessing unit processing to the images collected by the selected imagesensing unit; or selecting the nearest one distant from the first imagesensing unit and the second image sensing unit to the projecting unitinto the resting state, or stopping all modules of the image processingunit processing to the images collected by the selected image sensingunit, to further obtain, by the feature extracting module, some basisfeature unit pairs matched by a first basis feature unit and a secondbasis feature unit, to further calculate and obtain variation betweenthe basis feature unit pairs utilizing the feature deformation measuringmodule, for performing rough determination of the possible area positionthat the fingertip presents in the collected images of the image sensingunit selected to work, which comprises: If a distance from the finger atthe points of the deformation of the second basis feature unit to aprojecting plane is smaller than a predetermined first threshold value,determining a touch operation is occurred; if touching area is toolarge, then outputting coordinates of the touching area to the fingertipobtaining module, to obtain accurate touching coordinates; otherwise,directly outputting position coordinates of the corresponding touch areato the fingertip touch determining module, then turning to step S5; If adistance from the finger at the points of the deformation of the secondbasis feature unit to a projecting plane is bigger than a predeterminedsecond threshold value, determining a touch operation is not occurred,no process is performed by following, and returning to step S31; If adistance from the finger at the points of the deformation of the secondbasis feature unit to a projecting plane is between a predeterminedfirst and a second threshold value, outputting area coordinates of thefingertip evaluated in the image planes of the first image sensing unitand the second sensing unit to a fingertip obtaining module, thenturning to step S32; S32, finely searching a fingertip, in accordancewith the coarse positioning area of the finger obtained by step S31,this step includes searching, by the fingertip obtaining module, thefingertip in the area of the finger coarse positioning in the imagesreceived from and collected by the first image sensing unit and thesecond image sensing unit, and outputting the position coordinates ofthe fingertip obtained in the image planes of the first image sensingunit and the second image sensing unit to the fingertip touchdetermining module, then turning to step S4.
 12. The method of claim 10,wherein the step S2 further comprises scanning and obtaining aprojecting color predistortion parameter of the projecting plane andperforming color predistortion correction.
 13. The method of claim 10,wherein the step S2 further includes starting the feature extractingmodule to obtain system variable external parameters at regular time;that is, obtaining relative position and gesture information of theprojecting plane relative to the system, if the relative position andgesture is changed, re-calculating homography matrixes between the firstimage sensing unit and the second image sensing unit, and the projectingunit, and outputting again the changed system variable parameters to theinterface processing module, which updates the system variableparameters, and performs the processing of projecting shapepredistortion based on the changed system variable parameters, torealize tracking the projecting plane in real time.